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(57) ABSTRACT 

A surveillance system that includes a number of camera 
subsystems. Each camera subsystem includes a number of 
single lens cameras pointing in different directions which 
simultaneously capture images. Each camera subsystem 
covers a particular theater of interest. Each single lens 
camera captures an image of a particular region in a par- 
ticular theater. When an object of interest is identified, the 
system can follow the object as it moves between regions 
and theaters in a facility. When an object is being followed, 
the system can provide a view of the object as it transitions 
between regions covered different single lens cameras in a 
particular camera subsystem and as it transitions between 
theaters covered by different camera subsystems. 
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SECURITY CAMERA SYSTEM FOR In general, the term facility is used to describe the entire 

TRACKING MOVING OBJECTS IN BOTH area that is covered by the entire surveillance system. A 

FORWARD AND REVERSE DIRECTIONS facility covered by a surveillance system could, for example, 

be an airport terminal, it could be an entire shopping mall, 

RELATED APPLICATIONS 5 it could be a large office building, etc. The terra theater is 

__. -. . . . _ used to describe the area covered by a particular camera 

Tnis application is a non-provisional apphcaUon of pro- sllbsvsteni) the term k used t0 describe the area 

v*ional application serial No. 60/381,444 filed May 17, captured a single view and ^ term 

display area is used to describe the portion of a theater 
This application is also a continuation in part of the 10 displayed to an operator. 

following applications. < nmM « nn „ _ _ Each camera subsystem covers a particular theater of 

a) Co-pending application Ser. No. 10/003,399 filed Oct. 22, Each sin ^ e lens camera tures an . of a 

2001 entitled "Panoramic Movies which Simulate Move- particular region ^ a p arUcu i ar theater. Tlie system includes 
ment Through Multi-dimensional Space which is a con- multi k camera subsystems located at different positions in 
Unuation of application ^Ser. Nc . 09/310 715; filed May 12, 15 a fadlity ^ theatefS CQVcred b the subsystems have 
1999 which is now U.S^Pat No ^37 683 overlapping areas so that a person or object moving between 

b) Co-pending application Ser. No. 10/136 659 filed Apr. 30, meaters ^ contimiously under finance. The camera 

2002 entitled "Camera System With High Resolution Sllbsystems are pre f e rably located so that they cover 
Image Inside a Wide Angle View" which is a non- entrance and exit points to both the facility and to parts of 
provisional application based upon provisional applica- 2 o the facility 

tion No. 60/343,405 filed Nov. 19, 2001 and which is a m, • * a u n *u u , 

. f i- • The images captured by all the camera subsystems are 

continuation in part or applications. „ A U1 * _ * i * * a 

* made available to a central computer system. An operator 

1) Co-pending application Ser. No. 09/992,090 filed Nov. can select a particular display area for viewing at a particular 
16, 2001 entitled "Camera System With High Resolu- t j me ^he area selected for display by an operator can be 
tion Image Inside a Wide Angle View" 25 located in a region covered by one single lens camera, in one 

2) Co-pending application Ser. No. 09/994,081, filed Nov. camera subsystem or it can be an area which has parts of 
23, 2001 entitled "Camera System With High Resolu- regions covered by different single lens cameras which are 
tion Image Inside a Wide Angle View" part of a particular camera subsystem. 

Priority of the above applications is claimed and the When an object of interest is identified, the system can 

content of their specifications is hereby incorporated herein 30 follow the object as it moves between theaters in a facility, 

by reference. When an object is being followed, the system can provide a 

view of the object as it transitions between regions covered 

FIELD OF THE INVENTION different single lens cameras in a particular camera sub- 

Tlie present invention is related to digital camera systems 35 s ^ tem md ™ il ^nsitions between theaters covered by 

and more particularly to digital surveillance camera systems. different camera subsystems. 

All images captured by all the cameras are stored in a 

BACKGROUND OF THE INVENTION central data bank and one or more operator terminals are 

connected to the data bank for viewing images captured by 

U.S. Pat. No. 6,323,858 describes a camera system which me cameras. A moving object can be tracked as it moves 

includes a plurality of single lens cameras positioned on the *° f orwar d in time or the history of an object can be shown, 

faces of a cube. The single lens cameras simultaneously -p^ iSj the system can provide a series of images in real 

capture images that can be seamed into a panoramic image. X i m& as tbey are captured showing how an object is moving 

One can then select a view window into the panorama and through a facility. The system can also show the history of 

display a portion of the panorama on a monitor. an object's prior motion. That is, an object can be viewed by 

U.S. Pat. No. 6,337,683 describes a system for linking a 45 tracking the object backward in time through the stored 

plurality of images into a panoramic movie. A panoramic images. 

movie sequentially displays a view window into a series of BRIEF DESCRIPTION OF THE FIGURES 

panoramic images. With a regular movie the image which 

is presented to a viewer is fixed. With a panoramic movie, FIG. 1A is a diagram showing the locations of camera 

a viewer can select a particular direction of view and a view 50 subsystem in a facility such as an airport terminal, 

window showing that direction of view will be presented to FIG. IB is a diagram showing a camera subsystem camera 

the viewer. lens assembly. 

Co-pending application Ser. No. 10/136,659 shows a FIG. 2 is a system block diagram including a central 

camera system which includes a plurality of wide angle 55 computer system. 

single view cameras which capture a wide angle view of a FIG. 3 is a program flow diagram showing how a moving 

theater of operations and a moveable high resolution single object is followed as it moves within the views captured by 

lens camera which captures a selected area in the theater one camera subsystem. 

under surveillance. FIG. 4 is a program flow diagram showing how a moving 

object is followed as it moves between camera subsystems. 

SUMMARY OF THE PRESENT INVENTION FIG . 5 shows how an object can be identified in spaoe . 

The present invention provides a surveillance system that FIG. 6 shows an object moving between camera sub- 
includes a number of camera subsystems. Each camera systems. 

subsystem includes a number of single lens cameras point- FIG. 7 is a program flow diagram of the process of 

ing in different directions which simultaneously capture 65 detecting a target and building an attribute profile, 

images. The captured images together cover some portion of FIG. 8 is a program flow diagram of a program for 

the entire scene. tracking an object. 
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FIG. 9 is a program flow diagram of the program that of the camera systems may have more or less than five 

establishes tracks. lenses. Factors that influence the number of lenses used on 

FIG. 10 shows an alternate embodiment of the invention. each camera subsystem include, the desired resolution, the 

DETAILED DESCRIPTION OF A PREFERRED desired field of view > the 0051 etc - In the P rcferred embodi- 

EMBODIMENT 5 ment eac ^ °^ tne camera sub-systems CI to C9 are identical; 

_ * , . * , , however, in other embodiments, some of the camera sub- 

The preferred embodiment of the invention described ms haye different and/or jal features 

herein is a surveillance system in an airport terminal build- __, , _ ^„ , . 

ing. In a facility such as an airport terminal, people are ^ camera subsystems to C9 can be similar to the 

constantly moving from one part of the facility to other parts in cameras shown in issued U.S. Pat No. 6,323,858 Tne 

of the facility. When a suspicious person is identified, it is 10 f mera sh ™ n m P f • Na ^ 323 f a L slx lenses 

desirable to trace the person in a forward direction as the ^cing out from all sue faces of a cube With the present 

person moves between different parts of the facility. It is also mv !; ntlon > me ^m^ra head is suspended from the top of the 

desirable to trace the person's prior movement, that is, to fac f via arm , ' he L Qce > w " h * B Pf icular embodiment 

trace the person in a reverse time direction. The first embodi- no lens 15 needed on the t0 P of the cube * 

ment of the invention described herein provides a system As shown in FIG. 2, each of the camera subsystems CI to 

that can trace a person's movements as the person moves C9 provide a series of frames to a central computer 201. 

between theaters covered by different camera subsystems. Each frame (such as the illustrated frames Fl to Fx) includes 

The system can also trace a person's movement backward in five images and a header. The header has metadata which 

time. That is, when an operator focuses on an individual, the 2Q identifies the camera which captured the image and the time 

operator can trace movements by the individual backward in the ima g e was captured. The frames from each of the camera 

time to determine where the person originated and who the subsystems are stored at the central computer 201. A number 

person has previously contacted. The system will be of displays Dl to D3 are connected to the computer 201, 

described relative to tracing the movement of a person; either directly or over a network. The particular and selected 

however, it should be understood that any moving object can 25 display area from the captured images can be viewed on 

be traced and that a moving person is merely an illustration displays Dl to D3. 

of what can be traced. Any moving object can be identified Each of the cameras in each subsystems CI to C9 captures 

as an object of interest and traced in the same way that a images of a different region (i.e. a different region in the 

moving person is traced. theater covered by the camera subsystem). There is overlap 

The images captured by all the cameras are stored. When 30 in the regions covered by the single lens cameras in one 

a person of interest is identified, the movement of the person camera subsystem. There is also overlap in the theaters 

can be traced and viewed in both a forward and backward covered by the various camera subsystems. For example the 

direction single lens camera pointing to the right on camera C2, 

FIG. 1A illustrates an airport terminal facility. The termi- captures an image covering some of the same region cap- 
nal includes main entrance 103, ticket counter A, ticket 35 mred b y the lens pointing left on camera C3. 
counter B, merchandise shop A, merchandise shop B, a There are several modes of operation. FIG. 3 is a program 
boarding lounge A, a boarding lounge B, a boarding lounge flow diagram illustrating the operation of the system in a first 
C. Security barrier 102 has an entrance 101. The locations of mode of operation which only uses images from one camera 
the entrances to the boarding areas and shops are illustrated subsystem, FIG. 4 is a program flow diagram illustrating the 
by the short vertical lines. There are nine camera subsystems 40 operation of the system in a second mode of operation. In the 
in the facility. The camera subsystems are designated CI to second mode of operation the display area crosses a bound- 
CP. The camera subsystems cover overlapping theaters and ary between the theater of one camera subsystem to the 
they also cover the entrance and exit points of the facility in theater of a different camera subsystem. The system can 
general and the entrance and exit points to different parts of automatically move between the modes of operation as 
the facility. 45 required by the display areas which the operator or an 

FIG. IB shows the lens arrangement on a representative automated scanner selects to view. That is, either an operator 

camera subsystem. The camera head includes a plurality of can select an area or an automated scanner can select an area, 

lenses positioned on the five faces (of the six faces) of a In the first mode of operation (which only uses images 

cube. Only three of the lenses 416, 41c and 41e are visible captured by one camera subsystem and which is illustrated 

in FIG. IB. The other two lenses are on the faces opposite so in FIG. 3), a user at one of the displays Dl to D3, selects a 

lenses 41 b and 41c. Each camera subsystem captures five display area that the operator would like to observe. Each 

images, each image covers a particular region in the theater camera subsystem captures images which together cover at 

covered by that camera subsystem. For example, each single least a hemispheric panorama (one half of a sphere). An 

lens camera can have a 135 degree field of view. The operator can select to view a display area (i.e. a view 

combined 135 degree fields of view do not cover an entire 55 window) anywhere in this panorama. Note, the term "view 

sphere; however, the combined field of view does cover the window" is used relative to viewing panoramas, in order to 

regions of primary interest. The single lens cameras in each be consistent with the terminology conventionally used 

camera subsystem are simultaneously activated so that the relative to viewing panoramas. 

images captured by all the single lens cameras in a particular Initially, the operator selects a camera subsystem and the 

camera subsystem represent the same period in time. It is 60 operator is presented with a default view window from this 

best if the operation of all the single lens cameras in all the camera subsystem. Using a mouse or some other control 

camera subsystems operate simultaneously, but this is not device, the operator can move the display area to a new 

absolutely necessary. Likewise within each camera sub- location as indicated by block 302. As indicated by block 

system it is best if all the single lens cameras operate 303, a check is made to determine if the new location has 

simultaneously, but this is not absolutely necessary. 65 crossed a region boundary. That is, a check is made to 

In the preferred embodiment described, each camera determine whether the display area extends beyond the 

subsystem has five lenses. In other embodiments some or all region covered by one single lens camera. Stated still 
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another way, a check is made to determine whether the area 
covered by the view window covers both part of the region 
covered by a first single lens camera and part of the region 
covered by a different single lens camera in the same camera 
subsystem. S 

If a region boundary has not been crossed, the new area 
is displayed as indicated by block 304. If a region boundary 
has been crossed, the image from the other single lens 
camera is loaded, as indicated by block 305. The images are 
seamed as indicated by block 306 and the new view is 1° 
displayed. In an alternate embodiment, the images from each 
single lens camera in each camera subsystem are seamed 
into a panorama prior to being stored for viewing. However, 
by only seaming the images needed for a desired view 
window, the amount of computational time required is 15 
decreased. 

FIG. 4 shows a second mode of operation, tracking an 
object within and between camera subsystems. In the mode 
of operation illustrated in FIG. 4, a moving object is selected 
in the display area presented on one of the displays Dl to 20 
D3. The camera subsystem associated with the display 
containing the selected object is referred to in FIG. 4 as the 
current camera subsystem. Viewing consecutive frames of 
video, the view window then follows the motion of this 
moving object as illustrated by block 402. When the object 25 
moves to a point that it can no longer be seen by the current 
camera subsystem or is becoming very distant from the 
current camera subsystem (or would be better seen by 
another camera subsystem), the system determines which 
other camera subsystem would have a better view of this 30 
object as indicated by block 404. This can be done by 
consulting a table which stores the identity of the camera 
subsystem that will capture an image of a moving object as 
it leaves the view of each camera subsystem in each direc- 
tion. This camera becomes the current camera subsystem. 35 
The process of tracking the moving object then continues 
with this camera subsystem. 

FIG. 4 may be used to describe two different types of 
object tracking. The first type is tracking the object forward 4Q 
in time, for example in real time. In this instance, the phrase 
"consecutive frames" in Box 402 in FIG, 4 refers to video 
frames captured in natural recording sequence, moving 
forward in time. 

The second type of object tracking is tracking the object 45 
back in time, from archived frames. For example, if an 
operator observes a person entering boarding area A, the 
operator can trace the prior action of this individual back to 
the time the individual entered the terminal as observed by 
camera CI, In this case, the phrase "consecutive frames" in 50 
Box 402 in FIG. 4 refers to video frames captured in reverse 
order of natural recording sequence, i.e. moving backward in 
time. 

An object of interest (i.e. a person of interest) which is 
designated for tracking can be identified in a wide variety of 55 
ways. The person of interest can be designated by an 
operator who is continuously monitoring a particular site, 
such as main entrance 103 or by an official that notices 
something peculiar about a particular individual. The person 
of interest can be identified by an external system such as by 60 
a system that detects facial profiles and compares them to 
profiles in a data bank. A person of interest can be randomly 
selected for tracking on order to understand the pattern of 
movement in the facility. 

Tracking a person as the person moves between different 65 
regions covered by the different cameras in one subsystem 
or as a person moves between different theaters covered by 
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different subsystems uses the same techniques. These tech- 
niques utilize a profile of a moving object which the system 
assembles. When a person of interest is identified the system 
begins to detect and record the object's attributes such as 
velocity, cyclical movement profile, estimated height, color 
distribution, etc. Known types of logic can detect a wide 
variety of characteristics of an object from a series of images 
of the object. The particular characteristics detected in order 
to build the profile depend upon the application and the 
amount of expense appropriate for the particular system. The 
system can better and more accurately track moving objects 
if the system detects and records a profile with a large 
number of characteristics. The size, color and velocity of a 
moving object are relatively easy characteristics to detect 
and measure. Technology and software to detect and calcu- 
late other types of characteristics of an object are also known 
in the art. Furthermore, as an object moves between multiple 
different regions the profile of recorded characteristics can 
continue to expand and be refined. For example, both front 
and back profiles can be recorded. 

As explained above, in some embodiments, the operation 
of all the single lens cameras in all the camera subsystems 
is synchronized. That is, all the cameras capture images at 
the precise same instant. This facilitates tracking in that the 
pattern of motion of a particular object can be identified in 
images from different camera. If one looks at a moving 
picture of a group of people walking, the up and down 
motion of the people in the group is not synchronized. If two 
synchronized movie cameras record a particular group of 
people, the images from both cameras can be compared to 
identify the same pattern of up and down movement, and 
thus the same individual in both groups. Such a comparison 
of images can be done by known technology. 

Another important factor in tracking a person as the 
person moves from one region to another or from one theater 
to another is the precise location of the person at a particular 
moment when it is in the two images captured by two 
cameras. FIG. 5 illustrates an object 501 which can be seen 
by two single cameras 502 and 503. Cameras 502 and 503 
are in different camera subsystems. The cameras are 
attached to a ceiling 505 and the base of the object 506 is on 
the floor, hence, the height "h" is known. If the camera is 
moving from the region covered by camera 502 to the region 
covered by camera 503, the angle 502A is known, the 
distance between the cameras is known, hence, the angle 
503Acan be calculated. Thus from the location of the object 
501 in the image from camera 502, the location of the object 
in the image from camera 503 can be calculated. 

FIG. 6 illustrates how an object 600 is tracked as it moves 
between theaters covered by different camera subsystems. 
The tracking begins with camera 601. The object is detected 
and the system begins to detect and record the object's 
attributes such as velocity, cyclical movement profile, esti- 
mated height, etc. 

As shown in FIG. 6, the object 600 is tracked by camera 
subsystem 601 as it moves from position 1, to position 2 and 
to position 3. At position 3, the object is also in the theater 
of camera subsystem 602. The system knows that the object 
moved out of the theater covered by camera subsystem 601 
at a particular point. This tells the system that the object is 
now in the theater of camera subsystem 602. Camera Sub- 
system 602 detects the object using the profile assembled by 
camera 601. The profile is used to insure that the correct 
object is being tracked. The process continues when the 
object moves to position 5 where the tracking is handed off 
to camera subsystem 603. It is noted that while only three 
camera subsystems are shown in FIG. 6, the same operations 
can be expanded to any number of camera subsystems. 
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FIGS. 7, 8 and 9 are program flow diagrams of the 
program that does the object tracking. FIG. 7 relates to 
initially selecting an object for tracking. The process begins 
with the steps shown in FIG. 7. FIG. 8 is a block diagram of 
the main tracking process. FIG. 8 illustrates how attributes 
are recorded and the operations that take place when the 
object gets into a theater boundary where it is ready for 
hand-off to a different Camera subsystem. FIG. 9 shows how 
tracking is handed off from one camera subsystem to another 
camera subsystem. 

Each of the flow diagrams given in FIGS. 7, 8 and 9 will 
now be explained in more detail. As indicated by block 701, 
the tracking process begins when an object is selected for 
tracking. The selection can be done manually by an operator 
who points to a particular person on a screen and then clicks 
a mouse. Alternatively the selection can be by an automated 
detector which selects an object for tracking based on certain 
characteristics. However, in one of these ways if a target of 
interest is identified to block 703, a new tracking process is 
initiated as indicated by block 704. The system computes the 
direction vector from the camera subsystem to the object and 
the process of building a data base of the object's attributes 
begins. As explained above, the number of attributes mea- 
sured depends upon the particular application. The first 
tracking process is set as the Master tracking process as 
indicated by block 706. It is noted that in the area where an 
object is in the theater of two camera subsystems, there can 
be a master tracking process proceeding from the subsystem 
that started the process and a secondary tracking process 
from the camera subsystem to which the object is moving. 
Once the object is only being tracked by the second camera 
subsystem, that tracking process becomes the master track- 
ing process. 

The process continues from the bottom of FIG. 7 to the 
process shown in FIG. 8. FIG. 8 shows the common tracking 
logic that operates for all tracking processes in operation. As 
indicated by block 801 the first check made is to determine 
if the object is still moving (i.e. does it change position 
between image frames). If the object is moving the attribute 
profile is refined to reflect the motion as indicated by block 

803. Next as indicated by block 806, a check is made to 
determine if the object is near a boundary of the theater 
covered by the camera subsystem that is tracking the object. 
Next the system determines if the object is moving into a 
theater covered by another camera subsystem. It is possible 
that in some situations, not all areas are covered by a camera 
subsystem. 

Blocks 808, 810, 811, 812 and 807 determine if a tracking 
process should be started in an adjacent camera subsystem. 
It is noted that even if a new tracking process is initiated by 
block 812, the master tracking process proceeds until block 
805 indicates that the object has moved out of the theater of 
the camera subsystem that started the master tracking pro- 
cess and that process is terminated as indicated by block 

804. Thus block 812 both continues the tracking process via 
block 807 and it creates a new tracking process via the logic 
shown in FIG. 9. The process can also be terminated if the 
object stops moving and either the operator determines to 
stop the process or a timeout occurs. This test is shown by 
block 802. The decision block 802 indicates that the process 
does not continue if either one of two things happened: 
either a time out occurred or the operator has made a 
decision to terminate the process. 

FIG. 9 illustrates what happens when block 812 begins a 
new tracking process. The operations in FIG. 9 begin at 
point 901 when block 812 (see FIG. 8) begins a new tracking 
process. Block 908 indicates that the camera subsystem 
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which is "acquiring" the object is directed to the area where 
the object is entering its theater. This means that the system 
begins looking for the object in the image that covers that 
particular region of the theater. As indicated by block 907, 

5 the image is scanned to locate the target. The location of the 
object in an image can for example be initially estimated by 
considering the geometry as indicated in FIG. 5. Next, other 
characteristics of the object can be checked to insure that the 
correct object has been identified. That is, the target is 

1Q identified by looking for objects which have the attributes 
specified in the attribute profile. 

Alternatively an operator can visually locate the target. 
Block 906 is a test to determine if the object has been 
acquired. A time limit is set tested in blocks 905 and target 

15 observation terminates at block 902 if a preset time limit is 
exceeded. Block 903 is a check to determine if the previous 
camera is still tracking the object. If the object is still in the 
theater of the previous camera the result of this check will 
be "yes" and the new tracker process then proceeds to the 

2Q logic in FIG. 8; however, since the previous camera is still 
tracking the object, this will not be a Master tracking 
process. If the test at block 903 yields a "no" result, the 
process also proceeds to the logic in FIG. 8 as indicated by 
circle 911; however, in this situation the process will be the 

25 master tracking process. 

The above tracking process is described relative to track- 
ing a moving object as it moves from a theater covered by 
one camera subsystem to a theater covered by a different 
camera subsystem. The same process can be used to track an 

3 0 object as it moves between regions captured by different 
single lens cameras in the same camera subsystem. 
However, it is noted that the technology for tracking an 
object as it moves within the region presented by a single 
lens camera is well known. Thus, within each camera 

35 subsystem, the images may merely be seamed as an object 
moves from the edge of one region into another region and 
normal tracking algorithms may be employed to track the 
object as if there was only one image. This is termed 
on-the-fly seaming. In other embodiments, the images from 

4Q the various single lens cameras in each camera subsystem 
may be first seamed into a panorama, and then the tracking 
within a subsystem can be over a continuous panoramic 
image. 

It is noted that as described above, the tracking is done by 

45 cameras which acquire images which are then analyzed to 
detect objects. In alternate embodiments, the cameras could 
be replaced by other type of sensors. That is, for example 
instead of having a number of camera subsystems, one could 
have a number of acoustical, infrared, or radar detectors. 

50 Any type of sensor that can determine the attributes of a 
moving object can be used to implement the system in 
alternate embodiments. 

The tracking process, and the object profiles that track 
objects as they move across boundaries, must take into 

55 account the fact that as an object moves across a theater 
boundary, the view from one camera will be a front view and 
the view from the other camera may be a side or back view. 
Thus, the profiles must consider factors such as height or 
width of an object which are invariant from front to back. 

60 Furthermore, the physical location of an object, i.e. its x and 
y position, is generally used as a primary identification 
criterion, and other factors are used to verify that the correct 
object is being tracked. Where there is doubt about the 
identity of an object as it moves across a boundary, the 

65 operator is notified. 

Tracking moving objects is a relatively computationally 
intensive process. In order to take full advantage of all the 



04/21/2004, EAST Version: 1.4.1 



US 6,690,374 B2 

9 10 

computational power available in the computers that are part camera subsystem has only one wide angle, single lens 

of a surveillance system, when an object of interest is camera, the need to seam images captured by the camera 

identified, the system will normally begin to track an subsystem is eliminated. 

object's forward motion (unless the operator specifically In an alternate embodiment of the invention, the system 

decides to operate in another fashion). However, while the 5 includes two displays. Each display shows the images from 

object is being tracked in the forward direction, any avail- a particular camera subsystem. The operator can selectively 

able excess computer power can be used to begin tracking switch each display to any particular subsystem. In normal 

the object in the reverse direction. The results of the reverse operation, the two displays would show images from adja- 

tracking is stored for later use. In this way if an operator at cent camera subsystems. Under operator control the displays 

a later time asks the system to track the object in the reverse 1Q can be either in the form of a matrix of images from the 

direction, the results will immediately be available. individual single lens camera or they can be in the form of 

FIG. 10 is a diagram of another embodiment of the a view window into a panorama formed from the images 

invention. In this alternate embodiment at least one of the from a particular camera subsystem. With this system the 

camera subsystems in the system has a high resolution (i.e. operator can either be in complete control of tracking an 

narrower field of view) single lens camera that can be 15 object as it moves from one camera subsystem to another 

directed to specific an region of particular interest. As camera subsystem, or the operator can merely verify that a 

illustrated in FIG. 10, a high resolution lens is pointed at a program which tracks objects from one camera subsystem to 

mirror 951 and the region captured by this camera is another camera subsystem has correctly tracked an object 

controlled by moving the mirror 951 using motor controls across a theater or region boundary. 

952. That is, by moving the mirror 951, the single lens 20 The embodiments described above track a person as the 

camera (which faces up from the top of the unit) can be person moves about in a facility. It should be noted that the 

directed to capture a high resolution image of a selected above or various other embodiments can be used to track any 

area. The camera subsystem 950 shown in FIG. 10 is a moving object. The tracking can be done in a facility such 

camera of the general type shown in co-pending application a s an airport terminal, or it can be outside in an area covered 

Ser. No. 10/136,659 filed Apr. 30, 2002, the content of which 25 by camera subsystems. As used herein the term facility 

is hereby incorporated herein by reference. As used in the means any area covered by camera subsystems, 

embodiment described here, the camera has an orientation It will be underst ood by those skilled in the art that a wide 

that is different from the camera shown in the referenced variety of chan ges in form and detail can be made without 

co-pending application; however, the operation is similar. departing from the spirit and scope of the invention. The 

The camera subsystem 950 is suspended from the ceiling 30 SCO p e 0 f the invention is limited only by the appended 

of a facility by bracket 955. The subsystem has six single claims, 

lens cameras. FIG. 10 shows lens 961 and 962 on the sides \y e claim: 

of structure, lens 965 at the bottom of the structure, and the i, A surveillance system for a facility, said system includ- 
lens of the high resolution camera at the top of the structure. mg a p hi ra lity of camera subsystems, each camera sub- 
Two additional lenses (not visible in FIG. 10) are on the 35 system capturing images of a particular theater in said 
sides opposite lenses 961 and 962. The orientation of mirror facility, some of said theaters having overlapping areas, each 
951 is controlled by computer controlled motors 952. The camera subsystem including a plurality of single view 
details of how the mirror 951 can be controlled in order to cameras, each single view camera covering a particular 
direct the high resolution camera to particular region is reg j on j n a theater, said regions having overlapping areas, a 
described in the above referenced co-pending application. 4Q computer system for storing images captured by said single 
The high resolution image can be displayed either integrated v i ew cameras, a program for tracking an object as it moves 
with the display of the wide angle images or it can be shown across region boundaries in a camera subsystem and for 
on a separate display as described in the above referenced tracking an object as it moves across theater boundaries, 
co-pending patent applications. 2. The system recited in claim 1 including a computer 
When the high resolution camera is directed to a moving 45 program for identifying characteristics of an object and for 
object (for example an individual) that object is traced by the using those characteristics as the object moves across region 
high resolution camera in a manner similar to that explained or theater boundaries. 

with reference to FIGS. 6 to 9. Thus, not only is the object 3. The system recited in claim 1 including a computer 

tracked as it moves between camera subsystems, but the program for calculating the position of an object in one 

operator is also presented with a high resolution image of the 50 image and using said position to identify the same object in 

object. When the object moves to a theater covered by one another image. 

camera subsystem with a high resolution camera into 4. The system recited in claim 1 wherein each of said 

another theater covered by a different camera subsystem single view cameras capture images in synchronization so 

which has a high resolution camera, the high resolution that objects can be more easily tracked across region or 

camera in the acquiring camera subsystem is directed to the 55 theater boundaries. 

area where the object is located. Thus, an operator can 5. The system recited in claim 1 wherein said system 

continuously see a high resolution image of a moving object maintains a table indicating which theaters will have a view 
(except when it is obscured from the view of the high- of an object when the object moves out of a particular theater 

resolution by the main camera body), as the object moves in a particular direction. 

between theaters covered by different camera subsystems. 60 6. The system recited in claim 1 wherein said system 

It is noted that in the embodiment described above, each maintains a table indicating which regions will have a view 

camera subsystem includes a plurality of single lens cam- of an object when the object moves out of a particular region 

eras. It should be understood that in other embodiments of in a particular direction. 

the invention, each camera subsystem only includes one or 7. The system recited in claim 1, wherein portions of the 

two lenses. That is, the invention can be utilized with camera 65 panoramic images are seamed together in real time, 

subsystems which include fewer single lens cameras that 8. The system recited in claim 1, wherein the location of 

those shown in the above described embodiments. If each the tracked object in a first image captured by a first camera 
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subsystem is used to determine the location of the tracked 
object in a second image captured by a second camera 
subsystem. 

9. The system recited in claim 1, wherein the first and 
second images are high-resolution images. $ 

10. The system recited in claim 3, wherein the attribute 
profile includes a velocity of the tracked object. 

11. The system recited in claim 3, wherein the attribute 
profile includes a cyclical movement profile of the tracked 
object. ^ 1Q 

12. The system recited in claim 3, wherein the attribute 
profile is continuously refined as the tracked object moves 
through the environment. 

13. A method of tracking a moving object using a sur- 
veillance system including a plurality of camera subsystems, 
each camera subsystem capturing images of a particular 15 
theater in an environment having multiple theaters, each 
camera subsystem including a plurality of single view 
cameras, each single view camera covering a particular 
region in a theater, the method comprising: 

continuously tracking an object as it moves across region 20 
and theater boundaries; and 

generating panoramic images by seaming together single 
view images captured by the single view cameras, the 
panoramic images including single view images of the 
tracked object moving across region and theater bound- 25 
aries. 

14. A computer-readable medium having stored thereon 
instructions, which, when executed by a processor in a 
surveillance system including a plurality of camera sub- 
systems for capturing images of a particular theater in an 30 
environment having multiple theaters, each camera sub- 
system including a plurality of single view cameras, and 
each single view camera covering a particular region in a 
theater, cause the processor to perform the operations of: 

continuously tracking an object as it moves across region 35 
and theater boundaries; and 

generating panoramic images by seaming together single 
view images captured by the single view cameras, the 
panoramic images including single view images of the 
tracked object moving across region and theater bound- 40 
aries. 

15. A surveillance system, comprising: 

a plurality of camera subsystems, each camera subsystem 
capturing images of a particular theater in an environ- 
ment having multiple theaters, each camera subsystem 45 
including a plurality of single view cameras, each 
single view camera covering a particular region in a 
theater; 
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a rendering module for seaming together single view 
images into panoramic images; and 

a tracking module for tracking a moving object in a first 
panoramic image, identifying at least one attribute of 
the tracked object while it is being tracked in the first 
panoramic image, and tracking the moving object in a 
second panoramic image, wherein the attribute is used 
to distinguish between the tracked object and other 
objects in the second panoramic image. 

16. The system of claim 15, further comprising: 

a storage device for storing the panoramic images to 
facilitate tracking of the object backward in time. 

17. The system of claim 15, wherein the attribute is 
velocity of the tracked object. 

18. The system of claim 15, wherein the attribute is a 
cyclical movement profile of the tracked object. 

19. The system of claim 15, wherein the attribute is 
continuously refined as the object moves through the envi- 
ronment. 

20. A method of tracking a moving object in a series of 
panoramic images, comprising: 

tracking a moving object in a first panoramic image; 
identifying at least one attribute of the tracked object 

while it is being tracked in the first panoramic image; 

and 

tracking the moving object in a second panoramic image, 
wherein the attribute is used to distinguish between the 
tracked object and the other objects in the second 
panoramic image. 

21. A computer- readable medium having stored thereon 
instructions, which, when executed by a processor in a 
surveillance system, cause the processor to perform the 
operations of: 

tracking a moving object in first panoramic image; 
identifying at least one attribute of the tracked object 

while it is being tracked in the first panoramic image; 

and 

tracking the moving object in a second panoramic image, 
wherein the attribute is used to distinguish between the 
tracked object and other objects in the second pan- 
oramic image. 

22. The system of claim 15, wherein the attribute is at 
least one of a front and back profile of the tracked object. 
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